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ABSTRACT 

The  semi-Markov  shock  model  represents  a system  where  shocks 
occur  at  random  points  in  time  causing  random  magnitudes  of  damage 
in  such  a way  that  the  cumulative  damage  to  the  system  over  time 
is  a semi -Markov  process.  System  failure  can  occur  at  any  of  the 
shpck  times  and  the  probability  of  a failure  is  a function  of  the 
cumulative  damage.  An  optimal  'ontrol  limit  type  policy  has 

previously  been  derived  assuming  that  costs  are  Incurred  at  the 
replacement  time.  This  note  is  to  show  that  the  Inclusion  of  state 
dependent  maintenance  costs  does  not  significantly  affect  the  form 
of  the  optimal  control  limit. 
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1.  INTRODUCTION  AND  NOTATION 

The  optimal  control  limit  for  a semi-Harkov  shock  model  has 
been  derived  In  Feldman  (1976).  In  that  paper,  a very  simple  cost 
function  was  used.  The  purpose  of  this  note  Is  to  extend  the  results 
of  that  paper  to  Include  a general  cost  structure. 

Although  a complete  description  of  the  process  Is  given  In  the 
previous  paper.  It  will  be  repeated  here  for  ease  of  presentation. 

A A A A 

Let  the  Markov  renewal  process  (X,T)  » (X^,T^;  n=0,l,  . . . } be  a 
non-termlnatlng  process  with  state  space  E where  Tg,  Tj,  T2,  . . . 
denotes  the  Initial,  first,  second,  . . . shock  times  and  Xg,  Xj , 

A 

X2,  . . . denotes  the  cumulative  damage  at  the  respective  shock  times. 
(The  Initial  shock  time  Is  always  zero,  i.e. , Tq=0.)  The  state 
space  E Is  taken  to  be  a subset  of  the  non-negative  reals  or  the  non- 
negative  Integers.  Let  h(x)  denote  the  probability  that  the  system 

A A 

falls  at  time  T^  given  X^  x and  given  that  the  system  was  working 

A 

at  time  T^  The  function  h Is  assumed  to  be  non-decreasing,  and 

It  Is  assumed  that  there  exists  y - 0 and  some  n « 0,1,  . . . such 

that  h(y)>  0 and  P{  X I y I Xg  ■ x}  < 1 for  all  x 1 y.  (This  last 

assumption  Is  used  to  Insure  a finite  expected  failure  time.) 

Define  the  random  variable  L to  be  the  n such  that  T Is  the 

n 

failure  time  of  the  system;  thus, 

P{L  > n lXg,Xi,  . . .,X^}  - (l-h(Xi))  . . . (l-h(X^)).  (1) 


The  time  to  failure  Is  defined  by 


2 


Let  the  "death  state"  d be  a distinct  point  not  In  E and 
define  the  Markov  renewal  process  (X,T)  by 


XnO  - 


and 


T (w)  - 
n 


A 

X (<id  If  n < LCd)  > 


L If  n i L(bi) ; 

T (ad  If  n 1 L(u»), 
n 

■h*  If  n > L(u)) . 


(3) 


(4) 


The  semi-Markov  kernel  of  (X,T)  la  denoted  by  Q and  Is  defined  for 
X,  y ( E,  t i 0,  by 


Q(x,y,t)  - P{Xjly,  Tjit  1 X^-x}  (5) 

E P^{X,ly,.  Tjlt} 

" ^y,  T^lt,  L>1} 

- f Q(x,du,t)  (l-h(u)), 

Ju€P,y] 


where  Q Is  the  semi-Markov  kernel  of  (X,T)  and  should  be  a given 
quantity.  The  Markov  renewal  kernel  associated  with  Q Is  R and  Is 
defined,  for  x,  y fe  E,  t ^ 0,  by 

00 

R(x,y,t)  - I q"(x,y,t) 
n*0 


(6) 


where 


Q”^^(x,y,t) 


U£E 


Q(x,du,ds)Q  (x,y,t-s), 


8fe[0,t] 


The  Imbedded  Markov  chain  X will  have  transition  matrix  P and,  by 
an  abuse  of  notation,  its  potential  will  be  R;  thus 


and 


P(x.y)  = Q(x,y,t) 


R(x,y)  * lim  R(x,y,t)  « I p“(x,y) . 

n-0 


(7) 


(8) 


For  a fixed  control  limit  a,  let  denote  the  replacement 

time  and  let  A denote  the  Index  of  the  shock  time  at  which  replace- 
a 

ment  occurs;  that  Is, 


A ((j))  = L(u))  A Inf  {n:  X i a} 
a n 


(9) 


and 


T^(u)  * Cd). 
a 


(10) 


Equations  (3)  and  (4)  define  the  shock  model  and  equations  (9) 


and  (10)  define  the  replacement  policies.  In  the  next  section  the 
cost  structure  is  given  and  In  Section  3 It  Is  shown  that  previous 
results  apply  to  this  more  general  situation. 
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2.  THE  COST  STRUCTURE 


The  costs  used  in  Feldman  (1976)  were  a fixed  replacement  cost 
and  an  additional  Incremental  cost  Incurred  upon  failure.  In 
Feldman  (1977) , a general  replacement  cost  function  was  used  but  again 
no  maintenance  cost  was  considered.  Optimality  was  based  on  long 
run  average  cost  assuming  Instantaneous  replacement  with  an  identical 
system.  In  this  note,  both  a maintenance  and  a replacement  cost 
function  are  used.  Furthermore,  it  is  assumed  that  the  time  for 
replacement  is  a random  variable  denoted  by  o. 

Consider  the  renewal  process  formed  by  repeated  replacements 
of  identical  systems,  each  system  having  a lifetime  given  by  the 
distribution  of  and  a replacement  time  given  by  the  distribution 
of  0.  Let  the  cost  of  the  first,  second,  . . . replaced  system  be 
^1’  ^2'  * * * * C2,  . . .}  is  a sequence  of  Independent 

identically  distributed  random  variables.  If  is  the  number  of 
renewals  in  [0,t]  and  is  the  long-term  cost  per  unit  time,  then 


« lim  1[C,  + . . . + C„  ] 

“ t-+«  7 1 Nj. 


E[Cj^] 

ElTa+ol 


(11) 


Let  the  cost  of  replacement  be  given  by  the  function  f,  where 
f(x)  is  the  replacement  cost  if  the  cumulative  damage  is  x at  replace- 
ment time.  Let  the  maintenance  cost  be  determined  by  the  rate  function 
c(x) , where  c(x)  is  the  cost  per  unit  time  incurred  \diile  the  system 
remains  in  state  x.  Thus  the  expected  cost  per  system  is  defined 
by 

EICjl  - E(f«j  ) + [ c(X„)(I^j  - VI  . 

a n"0 


(12) 
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3.  OPTIMAL  REPLACEMENT 

The  main  trick  Is  to  substitute  the  maintenance  and  replace- 
ment cost  functions  by  a single  equivalent  replacement  cost  function. 
This  Is  not  a new  trick  and  was  done  by  Dynkln  (1963)  for  Markov 
processes  and  has  been  used  often  for  both  Markov  chains  and  Markov 
processes.  The  following  lemma  Incorporates  these  Ideas  for  Markov 
renewal  processes. 


(3.1)  LEMMA:  Let  (X,T)  be  the  Markov  renewal  process  defined 

by  equations  (3)  and  (4)  and  let  be  defined  by  equation  (10). 


Then 


A -1 
a 

E^[  I c(X  )(T  .,-T  )] 
X n n+1  n 

n=0 


R(x,dy)c(y)m(y) 

y f-E 


- E^[  R(X^  ,y)c(y)m(y)l , 
^ y ^ E a 


where  m(x)  Is  the  mean  sojourn  time  In  state  x;  or 


m(x) 


[l-Q(x,E,t)]  dt  . 

t>0 


Proof:  Let  be  an  arbitrary  non-negative  function  and  let 

R(^(x)  « I R(x,dy)  (^(y) . Then  for  the  Markov  chain  X, 

Jyf  E 

A-1 

Rf(x)  - E,t  I ♦(x  )]  + E [R.^(X.)]  (13) 

* n-0  " * ^ 

(see  ^Inlar  (1975,  page  201)).  For  ease  of  notation,  let 


J 


and  define  (x)  = c(x)m(x).  Now  note  that 


R<Kx)  - E^[  I c(X^)m(x)] 


X n 

n=0 


I E fc(X  )E  [W  |X  1] 

n*0  X n X n n 


I E [E  [c(X  )W  |X  ]] 
/.XX  n n n 
n*0 


= I E [c(X)*W  ] 
n-0  * " 


n=u 


Using  an  Identical  argument  on  the  other  two  terms  in  equation  (13) , 
the  result  of  the  lemma  Is  obtained. 


(3.2)  LEMMA:  For  a fixed  control  limit  a,  the  long-term  cost 

per  unit  time  Is  given  by 

f(x)  + I R(x,dy)[l«^(y)-^(y)] 

^ jy  , 

a ' 

E[a]  + I R(x,dy)m(y) 

h ^ t0,o] 


^(x)  - f(x)  - [ R(x,dy)c(y)m(y) 
•’y  E 


P?(x)  - f P(x,dy)?(y)  + f(A)[l-P(x,E)] 
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Proof:  The  denominator  of  equation  (11)  comes  from  Feldman  (1976, 
Proposition  2.10).  The  numerator  given  by  equation  (12)  uses  the 
relation 

E [f(X  )]  = f(x)  + [ R(x,dy)[Pf(y)-f(y)],  (17) 
* •'y<  a 


Lemma  (3.1),  and  the  substitution  given  by  (15).  The  proof  of 
equation  (17)  appears  in  Feldman  (1977)  but  since  that  proof  contains 
some  errors,  (17)  will  be  derived  below.  Using  a renewal  theoretic 
type  argument  and  the  fact  that  P(x,y)=0  for  x>y  (namely  the  system 
is  not  self  repairing) , It  follows  that 


Ejf(X^)J  = t(A)[l-P(x,E)]  + 


P(x,dy)f (y) 


y^a 


+ P(x,dy)E  [f(X^)] 
y<  a ^ 


y<  a 


Add  and  subtract 


ina 


R(x,dy){  P(y,du)f(u) 

^u^  a 

+ f(A)[l-P(y,E)]]. 

P(y,du)f(u),  rearrange  terms,  and  obtain 
,dy)Pf(y) 


I R(x,c 
•*y  < a 

- I f(u)|  R(x,dy)P(y,du) 
^u<a  -*y<a 


The  proof  is  now  completed  by  noting  that 
RP(x,u)  = R(x,u)  - IjQ^^j(x). 


1 


. J 
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(3.3)  THEOREM;  Let  (X,T)  be  the  Markov  renewal  process  defined 
by  equations  (3)  and  (4)  with  = x and  let  R(x,y)  be  an  absolutely 
continuous  function  with  respect  to  y for  each  x eE.  Then  the  optimal 
control  limit  policy  Is  the  a that  satisfies  the  following  equation: 

f R(x,dy){m(y)  [P^'(a)-?(a)  ]-m(a)  [P^(y)-?(y)  ] } 

Jy  [0,a] 

= m(a)f(x)-E[o-][P^(a)-f(a)]  . 

Proof:  Follows  from  Lemma  (3,2)  by  taking  a derivative  with  respect 

to  a. 

(3.4)  THEOREM:  Let  (X,T)  be  the  Markov  renewal  process  defined 

by  equation  (3)  and  (4)  with  the  state  space  the  set  of  non -negative 
integers,  let  Xq=1,  and  let  r(i,j)  = R(l,j)-R(l,j-1) . Then  the 
optimal  control  limit  policy  is  the  smallest  a such  that 

a-1 

^ r(i,k)  { m(k) [P?(a)-?(a) ]-m(a) [P?(k)-?(k) I } 
k=0 

i m(a)f(i)-E[aHP?(a)-f(a)]  . 

Proof:  Follows  from  Lemma  (3.2)  by  taking  differences  with  respect 

to  a. 

In  the  use  of  Theorem  (3.4),  it  should  be  pointed  out  that  the 
terms  r(l,j)  are  easy  to  compute  since  Q is  upper  triangular  (see 
Feldman  (1976,  Section  4)).  It  should  also  be  noted  that  equality 
is  Included  In  equation  (9)  so  replacement  occurs  when  the  cumula- 
tive damage  is  greater  than  or  equal  to  the  control  limit.  Finally, 
in  order  to  avoid  a common  computation  error,  it  might  be  helpful  to 
emphasize  that  the  state  space  E does  not  contain  the  state  A. 


•* 
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